When a follower comes up with an entirely empty disk, it can vote for any server to become leader which could result in data loss. For example with a R3 stream. Server A is leader and has stored all JetStream messages. Server B was part of quorum, but didn't apply all JetStream messages yet. Server C has stored some JetStream messages, but it didn't get the messages that had quorum on Server A and Server B.

If we'd shutdown Server B and reset the disk, we would violate stable storage which Raft relies on. If Server A goes down now, Server B can't become leader because it's empty (which is good), but Server B can vote for Server C to become leader. That would be bad, because we'd have lost some writes, and Server A would then have desynced.

By extension, scaling up of streams can be dangerous. Scaling up a R1 stream to R3 will work most of the time, but technically there's a very short time frame where the stream can be assigned on the two other servers, and the previous R1 leader immediately dies right after. Now the remaining servers that got assigned to be part of the R3 stream can vote for each other to become leader, which will lose the full stream contents.

This PR proposes to add protection and special handling when a server's log is empty.

• When a server's disk or log is fully reset, while its log is empty, it will vote using an "empty vote". This is a special kind of vote that the candidate will not count against normal quorum. Instead, empty votes can only be used to become leader if the candidate received votes from ALL servers. This protects against data loss even when only a single server hosts the data, as long as all servers can be available at the same time to sort out this reset issue.
• When an asset is scaling up, the Raft group will have ScaleUp: true set. This ensures servers that are new and part of the scale up go into observer mode. This ensures they can't become leader, but relaxes the above "empty vote" check. This ensures a candidate can still become leader based on normal quorum, and doesn't require votes from all servers.
• Also, when an asset is initialized the first time the "empty vote" check is also relaxed. Which ensures if you create a R3 stream/consumer, you can still rely solely on quorum and don't need all servers to be available at the same time.

Like I mentioned before, technically resetting a disk is already a violation of Raft, so we could call it a day there. But I feel like it's still a good thing to at least try to attempt to preserve the data, even in such a nightmarish scenario.

TL;DR

• meta/stream/consumer still require a quorum to store data
• meta/stream/consumer still require a quorum to be voted as leader
• only if a disk is fully reset, would meta/stream/consumer be put into a special mode that uses "empty votes", which ensures data can't be lost as long as at least 1 server has the data, and ALL servers are temporarily available at the same time to sort out this issue. After which we'd rely on quorum again, as usual.

Signed-off-by: Maurice van Veen [email protected]